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-J  The  Composite  Health  Care  System  (cues)  is  a  medical  information 
system  that  the  Department  of  Defense  is  developing  for  use  in  its  more 
than  690  medical  treatment  facilities~worldwide.  The  Congress  has 
capped  cues  life-cycle  costs  at  $1.6  billion.  Defense  is  required  by  law  to 
conduct  an  operational  test  and  evaluation  (ot&e)  of  cues,  perform  a 
cost/benefit  analysis,  and  report  the  results  to  the  armed  services  com¬ 
mittees  before  awarding  the  full-deployment  contract.  This  law  also 
requires  that  gao  monitor  ot&e  and  report  to  the  committees  within  30 
days  of  Defense’s  report. 

^This  report  is  one  in  a  series  dealing  with  Defense’s  acquisition,  develop¬ 
ment,  and  testing  of  this  system.1  Our  objectives  were  to  (1)  determine 
the  status  of  cues’  schedule,  performance,  cost,  and  benefits;  and  (2) 
identify  and  evaluate  changes  in  Defense’s  development,  testing,  and 
deployment  strategy.  We  conducted  our  evaluation  from  April  1990 
through  August  199/Mn  accordance  with  generally  accepted  govern¬ 
ment  auditing  standards.  Appendix  I  details  our  objectives,  scope,  and 
methodology.  1 


cues  shows  promise  of  greatly  enhancing  Defense’s  ability  to  manage 
patient-care  data,  but  Defense’s  recently  revised  strategy  for  testing  and 
deploying  the  system  is  unwise.  The  strategy  now  provides  for  a  deploy¬ 
ment  of  cues  with  incomplete  capabilities.  Defense  plans  a  March  1992 
decision  to  deploy  a  software  version  that  does  not  include,  as  originally 
planned,  either  the  capability  to  arc’  ive  and  retrieve  patient  records,  or 
an  efficient  method  for  entry  of  physicians’  orders.  The  capability  to 
archive  and  retrieve  patient  data  is  vital  to  cues’  operational  perform¬ 
ance  and  must  be  thoroughly  tested  and  included  in  the  software  version 
upon  which  a  deployment  decision  is  made. 


'See  tlie  Related  GAO  Ptoduets  sect  ion  at  the  end  of  this  reixirt. 
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Without  archiving,  system  response  time  slows  as  the  volume  of  data 
stored  on-line  reaches  disk-capacity  limits.  At  this  point,  with  no  way  to 
offload  data,  additional  disk  storage  and  related  equipment  are  required 
at  additional  cost.  Defense  has  not  yet  produced  any  analysis  estimating 
future  cost  growth  or  showing  how  such  costs  can  be  controlled  without 
archiving. 

Physicians’  entry  of  inpatient  orders  is  important  to  physician  accept¬ 
ance  of  the  system.  Since  it  is  also  crucial  to  realizing  a  significant  pur¬ 
ported  dollar  benefit,  it  is  not  yet  clear  that  a  version  of  cues  that  does 
not  include  physicians’  entry  of  inpatient  orders  will  be  cost  beneficial. 

Even  though  significant  operational  testing  issues  remain  unresolved 
and  a  credible  determination  of  the  system’s  costs  and  benefits  is  incom¬ 
plete,  Defense  has  expended  or  issued  delivery  orders  totaling  $12.9  mil¬ 
lion,  as  of  July  1991,  to  deploy  cues  at  77  medical  treatment  facilities 
before  the  completion  of  operational  testing.  This  deployment  does  not 
comply  with  Defense’s  own  policy  and,  further,  violates  legislation. 


Background 


cues  is  a  state-of-the-art,  integrated  medical-information  system 
designed  to  improve  the  timeliness,  availability,  and  quality  of  patient- 
care  data.  Currently,  Defense  estimates  the  life-cycle  costs  to  total  about 
$1.56  billion,  cues  will  replace  manual  and  automated  n  (formation  sys¬ 
tems  now  supporting  Defense  medical  treatment  facilities.  At  individual 
hospitals,  it  will  integrate  the  functional  work  centers  of  inpatient  and 
outpatient  care.  These  work  functions  include  physicians’  entry  of 
orders,  nursing,  and  dietetic  instructions  for  inpatient  care;  and  patient 
administration,  patient  appointment  and  scheduling,  laboratory,  phar¬ 
macy,  and  radiology  data  for  both  inpatient  and  outpatient  medical  care, 
cues  provides  a  greater  degree  of  patient  data  integration  than  is  cur¬ 
rently  available  in  commercial  hospital  systems.  The  integrated  patient 
record  is  intended  to  provide  physicians  with  immediate  access  to  all 
portions  of  a  patient’s  medical  record. 

In  March  1988,  Defense  awarded  a  contract  to  Science  Applications 
International  Corporation  (saic)  to  develop,  test,  deploy,  and  support 
cues.  Originally,  gt&e  for  cues  was  scheduled  for  completion  no  later 
than  September  1989.  As  of  July  15, 1991,  Defense  had  obligated  about 
$444  million  for  development  and  deployment  of  the  system.  Currently, 
completion  of  or&K  is  scheduled  for  March  1992.  The  project  will  then  be 
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submitted  to  the  Major  Automated  Information  Systems  Review  Com¬ 
mittee  (M4jsrc)  for  review  and  approval.  Defense  must  then  submit  the 
results  of  ot&e  and  a  cost-benefit  analysis  to  the  Congress. 


CHCS  Costs  and 
Benefits  Need  Further 
Analysis 


In  March  1990,  Defense’s  life-cycle  cost  estimate  for  chcs  was  increased 
above  the  $  1.1-billion  ceiling  established  by  the  Congress.  This  increase 
was  the  result  of  a  change  in  maisrc  guidance  that  defines  life-cycle 
cost(s)  and  time  frames.  Subsequently,  the  Congress  established  a  new 
$  1.6-billion  ceiling.  As  of  May  1991,  Defense’s  estimate  of  life-cycle  costs 
was  about  $1.66  billion.  However,  since  the  current  estimate  does  not 
include  the  impact  of  several  components  of  chcs,  it  is  subject  to  change. 
These  components  include  the  costs  to  archive  and  retrieve  patient  data, 
and  developing  a  more  streamlined  method  for  physicians’  entry  of 
orders. 


chcs  benefits  have  yet  to  be  quantified  by  a  convincing  analysis  and  doc¬ 
umentation.  In  March  1990,  Defense  expected  about  95  percent  of  the 
more  than  $2  billion  in  projected  benefits  to  result  from  reducing  the 
cost  of  the  Civilian  Health  and  Medical  Program  of  the  Uniformed  Ser¬ 
vices  (champus).  champus  pays  health-care  costs  for  active  duty  family 
members  and  retired  uniformed  services  members  and  their  dependents 
when  must  receive  medical  services  from  private  health-care  prov¬ 
iders.  Defense  assumed  that  chcs  would  make  medical  care  provided  by 
the  military  more  efficient  and  thus  reduce  the  number  of  people 
seeking  medical  services  from  private  health-care  providers.  In  March 
1990,  we  identified  weaknesses  in  Defense’s  benefits  study.2  Since  that 
report,  Defense  has  changed  the  moaner  in  which  it  portrays  chcs  bene¬ 
fits.  Currently,  Defense  is  claiming  that  deploying  chcs  will  permit  it  to 
offset  a  portion  of  the  expected  growth  in  future  health-care  costs. 

Defense’s  analysis  shows  that  health-care  costs  between  1986  and  1990 
increased  at  an  average  rate  of  8.4  percent  per  year.  Defense,  however, 
is  now  budgeting  for  later-year  health-care  cost  increases  of  only  4.4 
percent  per  year.  It  justified  this  lower  figure  by  claiming  that  chcs  will 
help  avoid  the  costs  that  make  up  the  difference  through  better  oper¬ 
ating  efficiencies  at  its  medical  treatment  facilities.  Defense  plans  to 
provide  evidence  to  support  claims  of  cost  avoidance  based  on  the 
results  of  operational  testing  and  evaluation,  which  are  currently  sched¬ 
uled  for  completion  next  March. 


'JMedical  ADP  Systems:  Composite  Health  Care  System:  Defense  Faces  a  Difficult  Task  (GAO/ 
lMTEC-!)(M2p  Mar.  15,  1990). 
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Defense  Now  Plans  to 
Deploy  CHCS  Without 
Essential  Capabilities 


In  March  1990,  we  reported  that  Defense  had  extended  ot&e  to  October 
1990  because  of  software-development  delays.3  At  that  time,  Defense 
planned  to  operationally  test  and  evaluate  a  fully  integrated  chcs  at  Fort 
Knox  and  6  of  its  12  test  sites  before  making  a  deployment  decision.  (A 
list  of  test  sites  is  shown  in  appendix  II.)  The  fully  integrated  system 
was  to  include  such  activities  as  patient  appointment  and  scheduling, 
physicians’  order  entry,  laboratory,  radiology,  pharmacy,  and  nursing. 
This  schedule,  however,  was  not  met. 

As  a  result,  Defense  extended  the  completion  date  for  ot&e  to  March 
1992.  But  much  more  importantly,  it  has  now  changed  the  strategy  for 
fielding  chcs,  Instead  of  making  a  decision  at  the  completion  of  ot&e  to 
field  a  fully  integrated  chcs  worldwide,  Defense  is  now  planning  to 
depioy  chcs  without  essential  capabilities.  According  to  Defense,  this 
decision  will  be  made  following  a  full  ot&e  of  the  capabilities  to  be 
fielded,  a  user-community  certification  that  the  version  of  chcs  to  be 
deployed  is  useful,  and  completion  of  a  supporting  cost-benefit  analysis. 
The  decision  will  receive  maisrc  review. 

The  deployment  decision  planned  for  March  1992  will  be  based  on  a  ver¬ 
sion  of  cues  software  that  does  not  include  two  important  capabilities: 
the  ability  to  archive  and  retrieve  patient  records,  and  a  streamlined 
method  for  physicians’  entry  of  patient  orders.  Because  of  difficulty  in 
designing  and  testing  these  capabilities,  they  have  been  deferred  and 
will  be  included  in  later  reviews  and  fieldings  of  chcs. 

Archiving — Archiving  is  the  ability  to  store  off-line  and  later  retrieve 
patient  data.  It  is  essential  to  effective  chcs  operations  and  is  still  under 
development.  Deploying  chcs  without  assurances  that  the  archiving 
capability  can  be  effectively  and  economically  incorporated  into  the 
system  could  ultimately  result  in  serious  operational  problems  and 
increased  costs.  Currently,  Defense  does  not  have  any  plans  to  test  this 
capability  at  any  chcs  ot&e  site  until  after  the  March  1992  deployment 
decision.  There  will,  however,  be  some  testing  of  archiving  and  retrieval 
in  the  contractor’s  laboratory. 

The  effort  involved  in  designing  and  implementing  an  effective 
archiving  capability  in  cues  is  complex.  It  is  technically  challenging,  in 
part,  because  chcs  software  is  highly  integrated,  relationships  among 
patient  information  are  complex,  and  a  nontraditional  file  structure  is 
used.  The  extraction  of  patient  information  from  this  file  structure  must 


3GAO/IMTEC-90-42,  Mar.  15, 1990 
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be  performed  in  a  flawless  manner  to  ensure  that  relationships  between 
all  segments  of  the  active  data  are  preserved  and  the  archived  data  can 
later  be  reassembled  when  needed  for  patient  care  or  medical  research. 
Before  such  a  complex  and  important  capability  is  deployed,  it  is  essen¬ 
tial  that  it  be  thoroughly  tested  in  a  representative  set  of  hospital 
environments. 

Operating  chcs  without  archiving  is  impractical.  Without  archiving, 
system  response  time  slows  as  the  volume  of  data  stored  on-line  reaches 
disk-capacity  limits.  Efforts  to  improve  the  response  time  can  be  time- 
consuming  and  disruptive,  and  eventually  ineffective,  as  even  more  data 
are  stored  on  disk.  At  this  point,  with  no  way  to  offload  data,  additional 
disk  storage  and  related  equipment  (controllers  and  faster  processors 
with  more  input/output  channels)  are  required.  As  patients  continue  to 
be  treated,  the  need  for  hardware  will  continue  to  grow.  The  costs  of 
such  growing  hardware  configurations,  however,  have  not  been  care¬ 
fully  analyzed  by  Defense.  For  example,  Tripler  Army  Medical  Center 
has  already  replaced  its  disk  drives  with  higher-density  models  and 
added  more  storage  units.  Defense  has  not  yet  produced  any  analysis 
estimating  future  cost  growth  or  showing  how  such  costs  can  be  con¬ 
trolled  without  archiving. 

Defense  plans  to  provide  an  archiving  capability  in  the  future.  However, 
given  that  archiving  is  technically  challenging  and  is  already  behind 
schedule,  and  that  software  projects  are  often  delivered  significantly 
behind  schedule,  it  is  not  judicious  to  deploy  chcs  hoping  that  its  open 
ended  cost  growth  will  be  controlled  by  a  capability  promised  in  the 
future. 

Physicians’  orders — Hospital  personnel  who  use  chcs  are  pleased  with 
the  system’s  performance  in  some  areas — primarily  outpatient  care 
functions.  However,  other  parts  of  the  system,  especially  the  process  by 
which  physicians  enter  inpatient  orders,  are  unacceptable  to  many  doc¬ 
tors.  This  unacceptability  is  primarily  due  to  the  way  physicians  have  to 
enter  both  conditional  or  complex  orders.  For  example,  treatment  of  a 
single  patient  can  require  a  physician  to  enter  data  through  the  key¬ 
board  to  bring  up  as  many  as  10  different  screens.  This  lack  of  accept¬ 
ance  was  evident  as  early  as  the  spring  of  1990  when  this  capability  was 
deployed  to  the  test  sites.  Since  then,  even  with  improvements,  only  5  of 
12  chcs  test  sites  have  activated  the  order  entry  functionality.  Although 
Defense  has  implemented  shortcuts  that  significantly  reduce  the  time  it 
takes  to  input  some  physician  orders,  much  remains  to  be  done  to 
streamline  this  process,  saic  and  Defense  are  aware  of  these  problems. 


I 
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They  are  working  toward  a  solution  that  is  commonly  known  as  one- 
line-order-entry  (oloe).  Defense  is  attempting  to  obtain  additional 
funding  to  expedite  the  development  and  testing  of  oloe. 

oloe  will  not  be  fully  developed  and  tested  as  part  of  the  current  ot&e. 
This  capability  is  critical  to  the  system’s  inpatient  functionality  and 
physician  acceptance  of  the  full  system.  It  directly  provides  about  10 
percent  of  the  system’s  total  benefits  according  to  Defense  estimates. 
Therefore,  it  is  unclear  whether  chcs  will  be  cost  beneficial  without 
OLOE. 


Premature 
Deployment  of  CHCS 
Violates  Defense 
Directives  and 
Legislation 


Defense  is  required  to  conduct  an  ot&e  of  all  major  automated  informa¬ 
tion  systems,  ot&e  of  an  automated  system  is  a  field  test  and  evaluation 
of  the  system  under  realistic  conditions.  The  primary  purpose  of  ot&e  is 
to  ensure  that  only  operationally  effective  and  suitable  systems  are 
delivered  to  the  ultimate  users  in  the  field. 

Defense’s  policy  requires  a  structured  process  for  maisrc  review  of 
information  on  systems  at  six  milestones  during  their  life  cycles.4  (A 
description  of  these  milestones  is  included  in  appendix  III.)  chcs  is 
nearing  milestone  III,  the  deployment  phase,  ot&e  provides  maisrc  with 
objective  data  on  system  performance  and  effectiveness,  and  as  such  is 
a  critical  input  in  deciding  whether  the  system  should  be  deployed. 


In  recent  years  significant  cost  increases  in  Defense  information  systems 
acquisitions — along  with  schedule  delays,  performance  shortfalls,  redi¬ 
rected  development  and  acquisition  strategies,  and  noncompliance  with 
regulations — have  led  the  Congress  to  question  Defense’s  ability  to 
manage  these  acquisitions  effectively .fi  More  specifically,  the  Congress 
has  questioned  whether  funds  are  being  obligated  for  systems  that  have 
not  successfully  completed  required  oversight  reviews.  To  preclude  this 
problem,  the  Congress  inserted  the  following  requirement  in  each 
Defense  appropriations  act  since  fiscal  year  1987: 


None  of  the  funds  appropriated  or  made  available  by  this  Act  may  be  obligated  for 
acquisition  of  major  automated  information  systems  which  have  not  successfully 
completed  oversight  reviews  required  by  Defense  Department  Regulations. 


4Life-Cycle  Management  of  Automated  Information  Systems  ( AIS),  Defense  Directive  7920.1,  and 
accompanying  Defense  Instruction  7920.2. 

5POP  Automated  Information  Systems  Experience  Runaway  Costs  and  Years  of  Schedule  Delays 
While  Providing  Little  Capability,  Report  of  the  House  Committee  on  Government  Operations, (HR 
101-382,  Nov.  20,  1989). 
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In  June  1988,  the  Office  of  the  Secretary  of  Defense  reemphasized  the 
statutory  language  to  require  that  the  head  of  each  Defense  component 

ensure  that  no  funds  are  obligated  for  the  acquisition  of  an  [automated  information 
system]  that  has  not  successfully  completed  an  appropriate  management  review  and 
obtained  milestone  approval  required  by  this  [directive  and  the  accompanying 
Defense  Instruction  7920.2]. 

Even  though  Defense  has  not  completed  ot&e  and  maisrc  has  not 
approved  chcs’  deployment,  Defense  expended  funds  to  deploy  chcs 
beyond  its  test  sites.  These  deployments  violate  the  preceding  Defense 
directives  and  statutory  restrictions.  As  of  July  1991,  Defense  had 
expended  or  issued  delivery  orders  totaling  $12.9  million  to  deploy  chcs 
at  77  medical  treatment  facilities  beyond  the  12  test  site  hospitals. 
Defense  has  already  allowed  the  Air  Force  to  deploy  a  version  of  chcs 
software  to  58  medical  facilities — 27  hospitals  and  31  clinics.  Deploy¬ 
ment  was  completed  in  April  1991  and  cost  the  Air  Force  about  $5.4 
million.  Defense  has  also  expended  or  issued  delivery  orders  totaling 
$7.5  million  to  begin  deploying  chcs  at  another  19  military  hospitals 
where  it  says  existing  systems  are  unreliable  and  experiencing  frequent 
and  lengthy  episodes  of  hardware  down  time.  Thus,  even  before  chcs 
has  completed  OT&E,  Defense  will  have  initiated  or  completed  installation 
of  a  version  of  chcs  at  58  hospitals,  more  than  one-third  of  the  155  hos¬ 
pitals  targeted  for  chcs. 


Conclusions 


cues  shows  promise  of  enhancing  the  quality  and  availability  of  Defense 
patient  care  data,  but  Defense’s  revised  strategy  for  deploying  the 
system  without  essential  capabilities  is  unwise.  Deploying  chcs  without 
archiving  is  impractical  because  this  function  is  critical  to  the  system’s 
operational  performance  and  the  establishment  of  a  credible  cost  esti¬ 
mate.  We  do  not  believe  any  deployment  decision  should  be  made 
without  an  archiving  function.  Regarding  physicians’  entry  of  orders,  it 
is  unclear  that  deploying  cues  without  it  will  be  cost  beneficial. 

Because  of  the  significant  financial  investment  needed  to  bring  chcs  into 
full  operation,  it  is  vital  that  there  be  an  ot&e  of  the  full  system  before  it 
is  deployed.  Defense’s  deployment  of  chcs  at  some  77  medical  treatment 
facilities  that  are  not  test  sites  is  premature  and  violates  existing  legisla¬ 
tion  and  Defense  directives. 
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Recommendations 


Agency  Comments  and 
Our  Evaluation 


We  recommend  that  the  Secretary  of  Defense  direct  the  Assistant  Secre¬ 
tary  of  Defense  for  Health  Affairs  and  the  military  departments  to 
refrain  from 

deploying  chcs  without  the  capability  to  archive  and  retrieve  patient 
data,  and 

further  deployment  of  chcs  until  completion  of  ot&e  and  the  perform¬ 
ance  of  a  cost-benefit  analysis  that  justifies  such  a  deployment. 


In  commenting  on  a  draft  of  this  report,  the  Department  of  Defense  dis¬ 
agreed  with  some  of  our  findings  and  one  of  our  recommendations.  Spe¬ 
cifically,  Defense  does  not  agree  with  our  finding  and  recommendation 
that  CHCS  should  not  be  deployed  without  the  capability  to  archive  and 
retrieve  patient  records.  It  also  believes  that  the  report  does  not  ade¬ 
quately  recognize  the  incremental  strategy  for  deploying  chcs.  Defense 
states  that  while  it  appreciates  the  work  we  have  done,  it  has  the 
responsibility  to  determine  the  required  functional  capabilities  for  each 
increment  of  deployment.  Finally,  Defense  disagrees  with  our  finding 
that  deployments  beyond  the  designated  test  sites  were  in  violation  of 
Defense  directives  and  statutory  restrictions. 

Although  Defense  does  not  agree  with  our  recommendation  that  chcs 
should  not  be  deployed  without  the  capability  to  archive  and  retrieve 
patient  data,  it  admits  that  this  capability  is  essential  to  system  opera¬ 
tions.  Defense  aH.  believes  that  there  is  low  risk  in  deploying  chcs 
without  the  capability  to  archive  and  retrieve.  We  do  not  agree  with  tins 
assessment.  For  example, .  ring  our  review  of  existing  chcs  test  sites, 
we  found  that  the  lack  of  archiving  was  contributing  to  operational 
instability  (fluctuations  in  system  response  times  because  of  the  volume 
of  data  stored  on-line)  and  increasing  costs  (additional  storage  had  to  be 
purchased  because  existing  capacity  had  been  exhausted). 

The  Defense  directive  on  test  and  evaluation  states  that  the  primary 
purpose  of  ot&e  is  to  ensure  that  only  operationally  effective  and  suit¬ 
able  systems  are  delivered.  The  directive  further  states  that  the  testing 
shall  be  accomplished  in  an  environment  that  is  as  operationally  real¬ 
istic  as  possible.  By  not  testing  this  capability  in  a  realistic  environment 
before  the  first  milestone  III  decision  is  made,  Defense  is  incurring  sig¬ 
nificant  risk  by  acquiring  a  system  that  may  not  meet  operational  needs 
and  may  not  be  cost-effective.  We  continue  to  believe  that  the  archive 
and  retrieve  capability  should  be  included  in  the  first  milestone  III 
deployment  decision,  not  only  because  it  is  essential  to  the  system’s 
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operation,  but  because  it  is  also  essential  to  Defense’s  ability  to  deter¬ 
mine  and  cont;  ol  the  cost  of  chcs  deployment  and  operations. 

Further,  it  is  important  to  note  that,  although  Defense  contends  that  it  is 
making  an  incremental  deployment  of  chcs,  the  decision  to  deploy  the 
first  increment  will  require  installation  of  nearly  all  the  operational 
hardware  and  software  ultimately  needed.  This  includes  full  physical 
and  electrical  site  preparation;  installation  of  central  processing  units, 
disk  drives,  and  terminals  for  all  hospital  operating  units;  communica¬ 
tion  systems;  full  chcs  software  packages;  and  user  training.  Subsequent 
incremental  deployments  will  consist  primarily  of  software  upgrades, 
more  and  higher  density  disks,  and  other  auxiliary  equipment. 

While  it  is  true  that  Defense  is  responsible  for  determining  the  required 
functional  capabilities  for  each  increment  of  chcs  deployment,  the  law 
requires  that  we  evaluate  this  program  and  inform  the  Congress  of  any 
significant  risks  that  Defense  is  taking  relative  to  chcs.  In  our  opinion, 
deploying  CHCS  without  the  archiving  and  retrieval  capability  is  a  signif¬ 
icant  cost  and  operational  risk. 

Defense  believes  that  all  chcs  deployments  have  been  in  compliance 
with  legislation  and  Defense  policy.  The  law,  however,  expressly  pre¬ 
cludes  the  obligation  of  appropriated  or  otherwise  available  funds  for 
the  acquisition  of  major  automated  information  systems  that  have  not 
successfully  completed  oversight  reviews,  as  required  by  Defense 
Department  Regulations.  Milestone  reviews  are  the  only  reviews 
required  by  Defense.  In  June  1988,  the  Office  of  the  Secretary  of 
Defense  reemphasized  that  statutory  language’s  connection  with  mile¬ 
stone  reviews  by  requiring  Defense  components  to  obtain  milestone 
approval  before  making  further  expenditures. 

Defense  further  states  that  a  maisrc  in-process  review  on  May  21, 1991, 
endorsed  the  incremental  deployment  of  cues.  The  deployments  of  chcs 
to  the  58  Air  Force  facilities  and  19  military  hospitals  not  included  as 
test  sites,  however,  are  not  related  to  the  incremental  deployment 
strategy  established  by  Defense  at  that  maisrc  review.  The  $12.9  million 
expenditure  by  Defense  to  deploy  chcs  to  these  facilities  occurred  prior 
to  the  May  maisrc  review.  Therefore,  we  continue  to  believe  that  these 
deployments  were  in  violation  of  Defense  directives  and  legislation. 

Detailed  Department  of  Defense  comments  and  our  evaluation  are  con¬ 
tained  in  appendix  IV. 
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We  are  sending  copies  of  this  report  to  the  Chairmen  of  the  House  and 
Senate  Committees  on  Appropriations;  the  Director,  Office  of  Manage¬ 
ment  and  Budget;  and  the  Secretary  of  Defense.  Copies  also  will  be  made 
available  to  other  interested  parties  upon  request. 

This  work  was  performed  under  the  direction  of  Frank  W.  Reilly, 
Director,  Human  Resources  Information  Systems,  who  can  be  reached  at 
(202)  275-4659.  Other  major  contributors  are  listed  in  appendix  V. 


Assistant  Comptroller  General 
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The  National  Defense  Authorization  Act  for  Fiscal  Year  1987,  as 
amended,  requires  that  gao  (1)  monitor  the  operational  test  and  evalua¬ 
tion  (ot&e)  phase  and  related  chcs  acquisition  activities,  and  (2)  submit 
a  report  to  the  Senate  and  House  Committees  on  Armed  Services  that 
evaluates  ot&e  results  and  Defense’s  contract  award  process  for  chcs’ 
full  production,  and  determines  whether  Defense  conducted  ot&e  at  a 
sufficient  number  of  sites  with  sufficient  software  in  operation  to  war¬ 
rant  a  full-deployment  decision.  The  act  requires  that  our  final  report  on 
ot&e  for  chcs  be  issued  30  days  after  the  Senate  and  House  Committees 
on  Armed  Services  receive  Defense’s  report  on  the  ct&e  results. 

Our  objectives  were  to  (1)  assess  the  status  of  chcs’  cost,  schedule,  and 
performance;  and  (2)  identify  and  evaluate  changes  in  Defense’s  devel¬ 
opment,  testing,  and  deployment  strategy.  In  conducting  our  review,  we 
examined  Defense’s  13  detailed  test  and  analysis  plans  for  ct&e; 
reviewed  Defense’s  most  current  (May  1991)  chcs  System  Decision 
Paper  and  supporting  documentation;  evaluated  the  monthly  progress 
reports  provided  to  Defense  by  the  chcs  contractor  through  August 
1991;  and  tracked  all  delivery  orders,  including  modifications  to  the 
delivery  orders,  which  Defense  issued  against  the  chcs  contract  through 
August  22, 1991. 

We  viewed  the  operation  of  chcs  at  5  of  the  12  operational  test  sites:  Ft. 
Knox,  Kentucky;  Eglin  Air  Force  Base,  Florida;  Naval  Hospital,  Jackson¬ 
ville,  Florida;  Naval  Hospital,  Charleston,  South  Carolina;  and  Tripler 
Army  Medical  Center,  Hawaii.  Fort  Knox  and  Tripler  serve  as  alpha  test 
sites  where  initial  testing  for  new  chcs  software  is  conducted.  We  also 
met  with  saic  officials  in  La  Jolla,  California  (the  prime  contractor),  and 
the  Defense  Medical  Systems  Support  Center  in  Falls  Church,  Virginia 
(the  program  office  managing  the  chcs  acquisition). 


In  addition,  we  met  with  Defense  and  contractor  officials  at  Malcolm 
Grow  Medical  Center  at  Andrews  Air  Force  Base,  Maryland,  which  is  an 
operational  test  site  for  the  deployment  of  limited  chcs  to  an  additional 
17  military  medical  treatment  facilities  beyond  the  ot&e  sites.  We 
obtained  information  on  the  condition  of  the  existing  automated  systems 
from  representatives  at  Malcolm  Grow  and  10  other  military  hospitals. 

We  worked  closely  with  senior  program  management  officials  to  (1)  dis¬ 
cuss  our  concerns  as  they  arose,  (2)  confirm  our  understanding  of  poten¬ 
tial  problems  and  their  implications  for  the  achievement  of  test 
objectives,  and  (3)  permit  the  officials  to  respond  to  our  observations. 
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We  conducted  our  evaluation  from  April  1990  to  August  1991  in  accor¬ 
dance  with  generally  accepted  government  auditing  standards.  We 
briefed  senior  program  management  officials  during  our  review  and 
incorporated  their  views  where  appropriate. 
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Tastsltas 

Hospital  bads 

Ireland  Army  Hospital 

Fort  Knox,  Kentucky 

184 

Tripler  Army  Medical  Center 

Honolulu,  Hawaii 

479 

Naval  Hospital 

Charleston,  South  Carolina 

184 

United  States  Air  Force  Hospital 

Eglin  Air  Force  Base 

Valpariso,  Florida 

145 

Naval  Hospital 

Jacksonville,  Florida 

178 

98th  General  Army  Hospital 

Nuernberg,  Germany 

142 

United  States  Air  Force  Regional  Hospital 

Sheppard  Air  Force  Base 

Wichita  Falls,  Texas 

135 

363rd  Medical  Group 

Shaw  Air  Force  Base 

Sumter,  South  Carolina 

40 

Eisenhower  Army  Medical  Center 

Fort  Gordon,  Georgia 

384 

United  States  Air  Force  Medical  Center 

Keesler  Air  Force  Base 

Bile...,  Mississippi 

295 

Walter  Reed  Army  Medical  Center 

Washington,  D.C. 

886 

Naval  Hospital  Camp  LeJeune 

Jacksonville,  North  Carolina 

170 
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Milestone  0:  The  purpose  of  milestone  0  is  to  determine  whether  to  pro¬ 
ceed  to  the  concepts-development  phase  on  the  merit  of  the  definition 
and  justification  of  a  mission  need.  A  mission-need  statement  is 
approved  at  milestone  0,  and  the  Defense  component  is  authorized  to 
initiate  the  concepts-development  phase  and  to  expend  resources  for  the 
activities  of  that  phase,  as  planned. 

Milestone  I:  The  purpose  of  milestone  I  is  to  select  the  best  program 
after  evaluating  functional  and  technical  alternatives  that  satisfy  the 
approved  mission-need  statement.  The  best  program  is  the  one  that  sat¬ 
isfies  the  mission  need  at  the  lowest  total  life-cycle  cost.  The  milestone  I 
approval  authorizes  program  management  to  initiate  the  design  phase 
and  to  expend  resources  for  the  activities  of  that  phase,  as  planned. 

Milestone  II:  The  purpose  of  milestone  II  is  to  validate  the  adequacy  of 
the  selected  automated  information  system  design  on  the  basis  of  com¬ 
pleted,  detailed  specifications.  Milestone  II  approval  authorizes  program 
management  to  initiate  the  development  phase  and  to  expend  resources 
for  the  activities  of  that  phase,  as  planned.  Milestone  II  approval  may 
include  authorization  to  test  and  evaluate  prototype  capabilities  at  a  set 
number  of  operational  installations. 

Milestone  III:  The  purpose  of  milestone  III  is  to  determine  whether  the 
completed,  comprehensively  tested,  and  operationally  capable  auto¬ 
mated  information  system  satisfies  the  mission  need  and  is  ready  for 
deployment.  Milestone  III  approval  authorizes  program  management  to 
begin  deployment  and  expend  resources  for  that  phase,  as  planned;  to 
begin  systems  operations  at  each  systems  site  upon  completion  of 
deployment  at  that  site;  and  to  transfer  systems  management  responsi¬ 
bility  from  the  program  manager  to  the  post-development  manager,  in 
accordance  with  approved  plans.  The  milestone  III  decision  memo¬ 
randum  identifies  the  milestone  IV  approval  authority  for  the  auto¬ 
mated  information  system. 

Milestone  IV:  The  purpose  of  milestone  IV  is  to  assess  post-deployment 
automated  information  systems  operations  and  to  approve  plans  for 
short-term,  post-deployment  systems  modernization.  Milestone  IV  occurs 
no  later  than  1  year  after  the  completion  of  systems  deployment.  Auto¬ 
mated  information  systems  post-deployment  management  submits  a  sys¬ 
tems  decision  paper  for  review  by  the  milestone  IV  approval  authority., 
Milestone  IV  approval  validates  that  the  mission  need  is  being  satisfied; 
operation  support  of  the  system  is  acceptable;  and  systems 
affordability,  performance,  and  benefits  are  within  acceptable  limits.  It 
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also  authorizes  post-deployment  management  to  expend  resources  for 
approved,  short-term,  post-deployment  systems  modernization. 

Milestone  V:  The  purpose  of  milestone  V  is  to  determine  if  the  existing 
automated  information  system  continues  to  satisfy  revalidated  mission 
needs,  requires  modernization,  or  should  be  terminated.  Milestone  V 
shall  occur  at  a  point  halfway  through  the  operational  life  of  the  system, 
or  not  later  than  4  years  after  milestone  IV,  whichever  occurs  first.  Mile¬ 
stone  V  approval  authorizes  post-deployment  management  to  program 
resources  for  long-term  systems  modernization  or  replacement  and  for 
initiation  of  the  concepts-development  phase.  A  fully  updated  and 
revalidated  mission-needs  statement  is  required  for  milestone  V 
approval. 
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Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


OFFICE  OF  THE  SECRETARY  OF  DEFENSE 
WASHINSTON,  D.C.  20301 


Me.  Ralph  v,  Carlone 
Assistant  Comptroller  General 
Information  management  and 
Technology  Division 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 


22  AU6  199V 


Dear  Mr.  Carlone: 

This  is  the  Department  of  Defense  (DoD)  response  to  the 
General  Accounting  Office  (GAO)  Draft  Report,  entitled  "MEDICAI. 
ADP  SYSTEMS;  Changes  in  Composite  Health  Care  System's 
Deployment  Strategy  Are  Unwise,"  dated  August  9,  1991  (GAO  Code 
510559),  OSD  Case  8780.  The  Department  agrees  with  the  GAO's 
observation  that  the  Composite  Health  Care  System  shows  promise 
of  greatly  enhancing  the  ability  of  the  Department  of  Defense  to 
manage  patient  care  data;  however,  the  Department  disagrees  with 
some  of  the  report  findings  and  one  of  the  recommendations. 

The  GAO  report  does  not  recognize  adequately  the  DoD 
incremental  deployment  strategy  for  the  Composite  Health  Care 
System,  as  endorsed  by  the  Major  Automated  Information  System 
Review  Council.  The  System  Decision  Memorandum  documenting  the 
May  21,  1991  In-Process  Review  reflects  that  strategy.  The 
Department's  actions  are  in  compliance  with  legislation,  are 
consistent  with  the  DoD  policies  for  life-cycle  management  of 
automated  information  systems,  and  will  continue  to  be  in 
compliance  with  legislation. 

The  DoD  appreciates  the  work  GAO  has  done  in  reviewing  the 
Composite  Health  Care  System  and  has  acted  positively  upon  many 
of  the  suggestions,  thereby  strengthening  the  program.  It  is, 
however,  the  Department's  responsibility  to  determine  the 
required  functional  capabilities  for  each  increment.  Such 
decision  making  is  necessary  to  meet  critical  information  system 
needs  and  to  begin  recouping  the  benefits  of  proven  system 
capabilities  as  soon  as  possible.  Deployment  decisions  will  not 
be  made  before  appropriate  cost,  benefit,  performance,  and  risk 
data  have  been  reviewed.  The  detailed  DoD  comments  on  the  report 
findings  and  recommendations  are  provided  in  the  enclosure. 
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Bnriqiik  Mendez,  Jr. ,  M.D.  Duane  p.  Andrews 

Assistant  Secretary  of  Defense  Assistant  Secretary  of  Defense 

(Health  Affairs)  (Command,  Control,  Communications 

and  Intelligence) 


Enclosure: 
As  Stated 
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GAO  DRAFT  REPORT  -  GAO/UfTKC-91-47  -  DATED  AUGUST  1991 
(GAO  CODE  510559)  OSD  CASE  8780 

■MEDICAL  ADP  SYSTEMS:  CHANGES  IE  COMPOSITE  HEALTH  CARE  SYSTEM'S 
DEPLOYMENT  STRATEGY  ARE  UNWISE" 

DEPARTMENT  OF  DEFENSE  COMMENTS 


FINDINGS 

FINDING  A:  Status_Of. The  Composite  Health  Care  System.  The 
General  Accounting  Office  reported  that  the  Composite  Health  Care 
System  is  a  state-of-the-art,  integrated  medical  information 
system  designed  to  improve  the  timeliness,  availability,  and 
quality  of  patient  care  data.  According  to  the  GAO,  the  DoD 
currently  estimates  the  life-cycle  costs  to  total  about  $1.56 
billion.  The  GAO  found  that  operational  test  and  evaluation, 
originally  scheduled  for  completion  by  September  1989,  is 
currently  scheduled  for  March  1992.  The  GAO  reported  that  as  of 
July  15,  1991,  the  DoD  had  obligated  about  $414  million  for 
development  and  deployment  of  the  system,  (pp.  1-2/GAO  Draft 
Report) 

DoD  Response:  Concur.  A  point  of  clarification  is,  however, 
indicated.  The  Composite  Health  Care  System  is  to  be  deployed  in 
increments.  By  March  1992,  operational  tost  and  evaluation  for 
the  first  increment  of  software  will  be  complete.  Additional 
operational  test  and  evaluation  will  be  conducted  on  subsequent 
increments  prior  to  their  deployment. 

FINDING  B:  The_Composlte  Health  Care  System  Costs  and  Benefits 
Need  Further  Analysis.  The  GAO  reported  that,  in  March  1990,  the 
DoD  life  cycle  cost  estimate  for  the  Composite  Health  Care  System 
exceeded  the  $1.1  billion  ceiling  established  by  the  Congress. 
According  to  the  GAO,  the  Congress  subsequently  established  a  new 
ceiling  of  $1.6  billion  for  the  System.  The  GAO  noted  that  the 
current  DoD  life-cycle  cost  estimate  for  the  System  is  about 
$1.56  billion.  The  GAO  observed  that  the  estimate  is  subject  to 
change,  since  the  current  estimate  does  not  include  the  costs  to 
(1)  archive  and  retrieve  patient  data,  (2)  configure  hardware  at 
sites  where  capacity  requirements  are  uncertain,  (3)  correct 
known  problems  with  system  software,  and  (4)  develop  a  more 
streamlined  method  for  physician's  entry  of  orders.  Also,  the 
GAO  found  that  the  system  benefits  have  yet  to  be  quantified  by  a 
convincing  analysis  and  documentation. 

The  GAO  referenced  a  March  1990  GAO  report  (OSD  Case  8277-A) ,  in 
which  it  reported  on  weaknesses  in  the  DoD  benefits  study.  The 
GAO  found  that,  since  that  report,  the  DoD  has  changed  -he  manner 
in  which  it  portrays  Composite  Health  Care  System  benefits. 
According  to  the  GAO,  the  DoD  claims  that  deploying  the  System 


Page  21 


GAO/IMTEC-91-47  CHCS  Deployment  Strategy 


Appendix  IV 

Comments  From  the  Department  of  Defense 


See  comment  1 . 


will  permit  it  to  offset  e  portion  of  the  expected  growth  in 
future  health  care  costs.  The  GAO  also  observed  that  the  DoD 
analysis  shows  health  care  costs  increasing  at  an  average  rate  of 
8.4  percent  per  year  between  1985  and  1990;  yet,  the  DoD  has  only 
budgeted  for  a  4.4  percent  per  year  health  care  cost  increase. 

The  GAO  reported  that  the  DoD  contends  deploying  the  Composite 
Health  Care  System  will  permit  it  to  offset  a  portion  of  the 
expected  growth  in  future  health  care  costs,  (pp.  4-5/GAO  Draft 
Report) 

DoD  Response;  Partially  concur.  Cost  and  benefit  analysis  is  a 
continuous  process  throughout  the  life-cycle  of  a  program  and 
updates  are  required  for  each  milestone  review.  Program 
life-cycle  cost  estimates,  including  those  cited  by  the  GAO,  are 
under  review  by  the  Office  of  Assistant  Secretary  of  Defense 
(Program  Analysis  and  Evaluation)  and  will  be  validated  prior  to 
the  Milestone  III  review.  The  Composite  Health  Care  System  cost 
estimate  was  verified  independently,  in  accordance  with  DoD 
regulations,  at  Milestone  II  and  will  continue  to  be  validated 
throughout  the  life  cycle  of  the  project.  The  cost  data  for  the 
System  estimates  are  based  on  experience  at  the  test  sites.  As 
with  any  model,  refinements  will  be  made  as  more  experience  is 
acquired. 

System  benefits  have  been  quantified  in  a  benefits  model  that  is 
also  being  evaluated  by  Program  Analysis  and  Evaluation  for 
Milestone  ill.  Most  of  the  benefit  factors  are  based  upon 
validated  benefits  from  independent  operating  capabilities  in 
support  of  laboratory,  appointment  and  scheduling,  radiology,  and 
pharmacy,  implemented  by  the  DoD  in  preparation  for  the  Composite 
Health  Care  System.  The  benefits  are  also  based  on  experience  at 
sites  with  early  hospital  information  systems.  The  model  will 
continue  to  be  updated  as  new  benefits  are  identified. 

In  addition  to  the  traditional  benefits  analysis  for  the  March 
1992  Milestone  III  review,  a  business  case  analysis  will  be 
presented  that  addresses  the  health  care  delivery  and  management 
processes  supported  by  the  System.  It  is  expected  that  analysis 
will  show  the  extent  to  which  benefits  of  the  Composite  Health 
Care  System  come  from  implementing  new  business  practices  and 
improved  ways  of  delivering  care  in  DoD  facilities,  which  are 
made  possible  by  the  System.  The  DoD  has  only  budgeted  for  4.4 
percent  per  year  health  care  cost  increases  because  deploying  the 
Composite  Health  Care  System  will  permit  it  to  offset  a  portion 
of  the  expected  growth  in  future  health  care  costs. 

A  further  point  of  clarification  is  that,  as  noted  in  the 
Department  of  Defense  response  to  the  March  1990  GAO  report,  the 
increase  in  the  ceiling  from  $1.1  billion  to  $1.6  billion  was 
requested  by  DoD  to  reflect  the  change  in  the  Major  Automated 
Information  System  Review  Council  guidance  that  defines 
life-cycle  costs  and  time  frames. 
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FINDING  C:  The  DoD  Plana  To  Deploy  the  Compoaita  Health  Ora 
System  Wlthout-JSssentlal  Capabilities.  The  GAO  pointed  out  that 
in  its  March  19!>0  report  (OSD  Case  8277-A),  it  reported  that  the 
DoD  extended  Orurational  Test  and  Evaluation  to  October  1990 , 
because  of  software  development  delays,  and  planned  to 
operationally  test  and  evaluate  a  fully  integrated  system  at  Fort 
Knox  and  at  six  of  its  12  test  sites  before  making  a  deployment 
decision.  The  GAO  found,  however,  that  because  the  schedule  was 
not  met,  the  DoD  extended  the  operational  test  and  evaluation 
completion  date  to  March  1992  and  changed  the  strategy  for 
fielding  the  system. 

The  GAO  observed  that  the  DoD  now  plans  to  deploy  the  system 
without  essential  capabilities,  such  as  (1)  the  ability  to 
archive  and  retrieve  patient  records  and  (2)  a  streamlined  method 
for  physician  entry  of  patient  orders,  because  of  design  and 
testing  difficulties.  According  to  the  GAO,  the  deployment 
decision  planned  for  March  1992  will  be  based  on  a  version  of 
software  that  does  not  include  those  capabilities  and  could 
result  in  serious  operational  and  financial  problems  in  the 
future.  The  GAO  concluded  that  operating  the  Composite  Health 
Care  System  without  archiving  is  impractical.  The  GAO  observed 
that  archiving  is  complex  and  technically  challenging  and  should 
be  thoroughly  tested  in  a  representative  set  of  hospital 
environments  before  deployme  c.  The  GAO  further  concluded  that 
without  the  physician  one-line-order  entry  capability,  which  will 
not  be  fully  tested  as  part  of  the  current  operational  test  and 
evaluation,  the  Composite  Health  Care  System  cost  benefits  are 
unclear,  (pp.  5-9,  p.  11/GAO  Draft  Report) 

DoD  Response:  Nonconcur.  Since  the  GAO  March  1990  report,  the 
DoD  changed  the  strategy  for  fielding  the  Composite  Health  Care 
System.  The  deployment  strategy  is  reflected  in  the  System 
Decision  Memorandum  dated  June  13,  1991,  documenting  the  May  21, 
1991  Council  In-Process  Review  of  the  System.  A  copy  of  the 
System  Decision  Memorandum  previously  was  provided  to  the  GAO. 

The  revised  strategy,  which  is  strongly  supported  by  the  Council, 
the  Assistant  Secretary  of  Defense  (Health  Affairs),  and  the 
Military  Department  Surgeons  General,  entails  the  incremental 
deployment  of  Composite  Health  Care  System  functionality,  which 
is  needed  to  meet  mission  requirements  and  is  ready  for  Council 
review.  Each  increment  will  be  tested  fully  in  an  operational 
test  and  evaluation  environment.  The  results  will  be  presented 
to  the  Operational  Test  and  Evaluation  Review  Group  and  then 
reviewed  by  the  Office  of  the  Secretary  of  Defense  (Director, 
Operational  Test  and  Evaluation).  Those  increments  containing 
significant  additional  capabilities  will  be  forwarded  for  Council 
review  prior  to  deployment  beyond  test  sites. 

Archiving  and  retrieving  of  patient  records  is  essential  to 
System  operations  and  will  be  tested  thoroughly.  In  fact, 
development  of  the  archive  and  retrieve  capability  has  been 
completed  and  testing  has  already  begun.  Approximately  seven 
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months  of  comprehensive  testing  of  the  archive/retrieve 
capability  will  occur,  as  well  au  an  independent  Early 
Operational  Assessment,  with  the  results  submitted  to  the  Major 
Automated  Information  System  Review  Council  prior  to  the  first 
Milestone  III  review.  There  will  be  sufficient  evidence  obtained 
through  testing  to  ensure  that  the  archiving  capability  will  be 
incorporated  effectively  and  economically  into  the  system. 
Finally,  risk  appears  to  be  low  because  experience  has  shown 
saturation  is  not  reached  until  18-24  months  after  activation. 
Based  upon  the  post  Milestone  III  deployment  plan,  the  earliest 
that  saturation  would  be  reached  is  March  1994  and  archive/ 
retrieve  capability  will  be  available  for  full  deployment  by 
early  calendar  year  1993.  Any  deployment  decision  will  occur 
after  a  complete  operational  test  and  evaluation  and  include 
consideration  of  the  risks,  costs  and  schedules  for  fielding  this 
capability. 

The  Department  is  actively  pursuing  completion  of  the  one-line- 
order-entry.  Full  inpatient  order  entry  presently  is  operating 
at  Ireland  Army  Community  Hospital,  Fort  Xnox,  Xentucky  and 
Tripler  Army  Medical  Center,  Honolulu,  Hawaii  and  partial 
inpatient  order  entry  is  running  at  the  beta  test  sites.  Three 
initiatives  are  in  process  to  support  the  effort.  1)  Additional 
on-site  program  resources  have  been  dedicated  to  Tripler  and  Fort 
Xnox  to  enhance  the  process  of  understanding  the  necessary  design 
changes  and  improvements  as  recommended  by  the  "Report  -  National 
Defense  Authorization  Act  for  Fiscal  Years  1992-1993  on  H.R. 
2100."  2)  The  professional  clinical  user  community  is  leading 

the  redesign  effort.  Approximately  40  experienced  health  care 
professionals  from  all  three  Services'  military  medical  treatment 
facilities  met  in  July  1991  to  discuss  and  document  the  necessary 
changes.  3)  The  Office  of  the  Assistant  Secretary  of  Defense 
(Health  Affairs)  has  requested  $3  million  from  the  Corporate 
Information  Management  Program  for  acceleration  of 
one-line-qrder-entry  implementation. 

The  report  states  that  early  operational  test  and  evaluation 
found  that,  in  some  instances,  Composite  Health  Care  System 
physician  inpatient  order  entry  took  from  25  to  40  minutes  longer 
than  handwriting  orders.  It  is  inappropriate  to  compare  the  time 
for  a  physician  to  write  an  order  with  the  time  required  to  enter 
that  order  on  the  computer  via  the  System  order  entry  module.  In 
the  Composite  Health  Care  System,  at  the  completion  of  the  order 
entry  process,  the  order  is  checked,  validated,  and  transmitted 
to  the  appropriate  department.  In  the  manual  process,  the 
handwritten  order  still  has  to  be  transcribed  to  other  forms 
(e.g.,  laboratory  transmittal  slips,  nursing  due  lists,  etc.),  or 
read  and  interpreted  by  other  health  care  providers  (e.g., 
pharmacists  and  nurses) .  Additionally,  the  order  has  to  be 
hand-carried  or  telephonically  transmitted  from  the  point  of 
origin  to  its  final  destination.  For  fair  comparison,  the  total 
time  for  the  manual  process  and  the  comparable  Composite  Health 
Care  System  process  must  be  considered. 
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In  early  1991,  an  actual  comparison  was  conducted  between  Tripler 
Army  Medical  Center,  Honolulu,  and  the  national  Naval  Medical 
Center,  Bethesda.  The  time  required  to  process  a  standard  13 
order  set  at  Tripler  Army  Medical  Center,  using  the  Composite 
Health  Care  System  was  compared  to  the  time  required  to  process 
the  same  orders  at  Bethesda  using  handwritten  orders.  It  took 
less  than  2.5  minutes  to  enter  and  process  this  standard  order 
set  using  the  Composite  Health  Care  System  at  Tripler.  The  time 
required  to  manually  process  these  same  orders  was  shown  to  vary 
from  7  to  17  minutes  at  Bethesda.  Thus,  not  only  is  the  time 
required  to  enter  admissions  orders  shorter  than  the  30-45 
minutes  cited  in  the  GAO  report,  but  in  addition,  they  are  fully 
validated  and  transmitted  as  part  of  inpatient  order  entry.  The 
majority  of  inpatient  orders  are  processed  more  effectively  using 
the  Composite  Health  Care  System.  The  small  subset  of  complex 
inpatient  orders,  e.g.,  intensive  care  unit  orders,  that  do 
require  more  physician  time  are  being  addressed  and,  as  discussed 
above,  efforts  are  underway  to  streamline  the  processing. 

The  GAO  concluded  that  it  is  unclear  whether  the  Composite  Health 
Care  System  will  be  cost  beneficial  without  one-line-order-entry 
in  the  first  increment.  It  is  due  to  the  critical  need  of 
certain  Composite  Health  Care  System  functions  required  to  offset 
growing  cost  and  health  care  delivery  and  management  problems 
that  the  decision  was  made  to  incrementally  develop  and  deploy 
the  Composite  Health  Care  System.  While  one-line-order-entry  may 
account  for  a  small  percentage  of  the  System  benefits,  the 
majority  of  the  benefits  will  be  achieved  by  functionality 
contained  in  the  first  increment,  as  implemented  at  the  medical 
treatment  facilities.  These  functions,  which  include  integrated 
ancillary  support  to  the  areas  of  patient  administration,  patient 
appointing  and  scheduling,  pharmacy,  laboratory,  and  radiology, 
along  with  flexible  level  of  results  reporting  capability  for 
direct  health  care  professionals,  are  immediate  requirements. 

The  decision  to  deploy  the  Composite  Health  Care  System 
incrementally  is  a  low  risk  pathway,  driven  by  the  recognition 
that  medical  treatment  facilities  desperately  need  this  Composite 
Health  Care  System  functionality.  Many  are  without  these 
essential  functions  or  have  costly,  undersized  and  aging 
automated  systems  that  require  immediate  replacement. 

In  addition  to  these  cited  benefits,  the  advantages  derived  from 
deployment  of  the  majority  of  the  Composite  Health  Care  System 
after  a  March  1992  review  include:  the  elimination  of  duplicate 
data  entry;  military  health  services  system-wide  utilization  of 
standard  data  elements  and  codes;  the  installation  of  standard 
hardware  architecture  which  can  easily  grow  to  accommodate 
increases  in  both  workload  and  functionality;  and  more  time  to 
properly  address  the  complex  issues  surrounding  one-lino-order- 
entry.  The  modular  deployment  philosophy  of  the  System  does  not 
detract  from,  but  complements  the  capability  to  deploy  fully, 
since  the  full  Composite  Health  Care  System  builds  on  an 
integrated  patient  data  base,  which  is  installed  with  the  first 
increment  of  software. 
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gIMP.IBfi.Iti  Premature  Deployment  of  the  Composite  Health  Care 
Svatem  Violates  Defense  Directives  and  Legislation.  The  GAO 
reported  that  the  DoD  la  required  to  conduct  an  operational  test 
and  evaluation  of  all  major  automated  information  systems  to 
ensure  their  suitability  and  effectiveness  under  realistic 
conditions.  The  GAO  explained  that  DoD  policy  requires  a 
structured  process  for  the  Major  Automated  Information  System 
Review  Council  to  review  information  on  systems  at  six  milestones 
durinq  their  life  cycles.  The  GAO  noted  that  the  Composite 
Health  Care  System  is  nearing  a  Milestone  III  deployment 
decision,  and  that  operational  testing  and  evaluation  provides 
data  critical  to  the  Review  Council  deployment  decision.  The  GAO 
reported,  however,  that  recent  significant  cost  increases  in  DoD 
information  systems  acquisitions  have  led  the  Congress  to 
question  (1)  the  DoD  ability  to  manage  the  acquisitions 
effectively  and  (2)  whether  funds  are  being  obligated  for  systems 
that  have  not  completed  the  required  oversight  review 
successfully.  The  GAO  explained  that,  to  preclude  the  problem, 
the  Congress  has  inserted  language  in  each  DoD  appropriations 
bill  since  FY  1987,  and  stated  that  funds  are  not  to  be  obligated 
for  major  automated  information  systems  acquisitions  that  have 
not  completed  successfully  the  oversight  reviews  required  by  DoD 
regulations.  The  GAO  observed  that,  in  June  1988,  the  Office  of 
the  Secretary  of  Defense  reemphasised  the  statutory  language  and 
required  component  heads  to  ensure  that  no  funds  are  obligated 
for  an  automated  information  system  if  it  has  not  completed  an 
appropriate  management  review  successfully  and  obtained  milestone 
approval  required  by  DoD  Directives. 

The  GAO  found  that,  although  Composite  Health  Care  System 
operational  test  and  evaluation  has  not  been  completed  the 
Ma j jr  Automated  Information  Systems  Review  Council  has  not 
approved  system  deployment,  the  DoD  has  expended  funds  to  deploy 
the  System  beyond  its  test  sites.  The  GAO  pointed  out  that  the 
DoD  issued  purchase  orders  totaling  $13.2  million  to  deploy  the 
System  at  76  medical  treatment  facilities  and  an  additional  $7.8 
million  to  begin  deploying  the  System  at  another  18  military 
hospitals  experiencing  problems  with  their  existing  systems.  The 
GAO  concluded  that  those  deployments  violate  DoD  Directives  and 
statutory  restrictions.  The  GAO  explained  that  before  the  System 
has  completed  operational  test  and  evaluation,  the  DoD  will  have 
initiated  or  completed  installation  of  a  version  of  the  system  at 
57  hospitals — more  than  one-third  of  the  155  hospitals  targeted 
to  receive  the  system.  The  GAO  also  concluded  that  an 
operational  test  and  evaluation  of  the  full  system  is  vital 
before  it  is  deployed,  (pp.  9-12/GAO  Draft  Report) 

DoD  Response;  Nonconcur.  The  Department  of  Defense  is  in  full 
compliance  with  applicable  Defense  guidance  and  legislation.  The 
Major  Automated  Information  System  Review  Council  reviews  are 
being  held,  as  required.  Attachment  III  to  the  GAO  report  quoted 
Defense  Instruction  7920.2,  in  which  the  GAO  noted  '  at  the 
milestone  reviews  required  during  the  life  cycle  o.  a  system 
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"represent  the  minimum  set  of  decision  points  requiring  direct 
senior  management  involvement  in  an  automated  information  system 
program."  The  applicable  statute  does  not  require,  nor  has  it 
been  DoD  policy,  that  oversight  reviews  be  limited  only  to 
milestone  reviews.  Senior  management  involvement  and  decisions 
are  also  made  at  times  other  than  Milestone  reviews.  For 
example,  the  21  May  1991  Review  Council  In-Process  Review 
endorsed  the  incremental  deployment  of  the  Composite  Health  Care 
System.  The  System  is  receiving,  and  will  continue  to  receive, 
considerable  more  than  the  minimum  Review  Council  oversight. 

Deployments  to  date  are  consistent  with  Review  Council 
authorization  and  existing  legislation.  Applicable  DoD 
regulations  specifically  provide  for  conditional  approvals.  The 
Composite  Health  Care  System  program  office  has  not  deployed  any 
software  without  having  explicit  authority  to  do  so  from  the 
appropriate  DoD  Information  Resources  Management  official.  In 
addition  to  the  12  operational  test  and  evaluation  sites, 
software  was  deployed  to  Malcolm  Grow  Medical  Center  at  Andrews 
Air  Force  Base,  for  certification  testing  of  the  Patient 
Appointing  and  Scheduling  module  in  compliance  with  congressional 
direction.  Prior  to  legislation  limiting  the  deployment  of  the 
Composite  Health  Care  System,  a  sound  business  decision  was  made 
to  deploy  the  Standard  Appointment  and  Scheduling  System  to  58 
Air  Force  sites  (27  hospitals  and  31  clinics).  The  Air  Force 
Surgeon  General  documented  an  immediate  critical  need  for 
automated  patient  appointing  and  scheduling  support  and  Defense 
had  available  government  owned  software  which  was  capable  of 
supporting  this  requirement.  The  software,  however,  represents 
only  one  of  nine  functional  areas  of  the  total  system. 

Subsequent  legislation  (Fiscal  Year  1991  National  Defense 
Authorization  Act)  limiting  System  deployment,  was  implemented 
promptly.  The  legislation  resulted  in  reexamination  of  two 
delivery  orders  for  site  deployment,  which  were  canceled.  Thus, 
depending  .upon  the  outcome  of  the  Patient  Appointing  and 
Scheduling  certification,  only  the  Patient  Appointing  and 
Scheduling  module  of  the  Composite  Health  Care  System  would  be 
activated  at  the  27  hospitals  with  the  Standard  Appointment  and 
Scheduling  System  and  at  the  Patient  Appointing  and  Scheduling 
certification  site.  The  full  System  is  being  activated  at  the  12 
test  sites.  No  further  deployment  will  occur  prior  to  completion 
of  operational  test  and  evaluation.  The  Department  of  Defense 
intends  to  remain  fully  compliant  with  its  guidance  and 
legislation. 

Expenditures  to  date  to  deploy  the  Composite  Health  Care  System 
need  to  be  clarified.  The  GAO  stated  that  as  of  April,  1991, 
deployment  to  Air  Force  sites  was  completed  at  a  cost  of  about 
$5.4  million.  In  fact,  the  deployment  of  the  Standard  Appointing 
and  Scheduling  System  was  completed  in  April  1991,  at  a  cost  of 
$3.8  million.  The  GAO  also  stated  that  Defense  had  expended  or 
issued  purchase  orders  at  another  18  military  hospitals  for  a 
total  of  $7.8  million.  The  Department  issued  purchase  orders  for 
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a  total  of  $1.7  million  to  conduct  site  surveys  at  18  sites. 
Additionally,  $3.2  million  has  been  expended  to  install,  operate 
and  maintain  the  Patient  Appointing  and  Scheduling  module  at  the 
Malcolm  Grow  Medical  Center  at  Andrews  Air  Force  Base  for 
certification  testing  in  compliance  with  congressional  direction. 


•  *  *  *  * 


RECOMMENDATIONS  TO  THE  DEPARTMENT  OF  DEFENSE 


RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary  of 
Defense  direct  the  Assistant  Secretary  of  Defense  for  Health 
Affairs  and  the  Military  Departments  to  refrain  .com  deploying 
the  Composite  Health  Care  System  without  the  capability  to 
archive  and  retrieve  patient  data.  <p.  12/  GAO  Draft  Report) 

DoD-Reapgnge:  Nonconcur.  It  is  premature  to  make  a  decision 
regarding  the  deployment  of  the  archive/retrieve  capability  until 
all  data  are  presented  to  the  Major  Automated  Information  System 
Review  Council.  At  the  first  Milestone  III  review,  based  on 
information  on  costs,  benefits,  performance,  risks,  and  an  early 
operational  assessment,  the  Review  Council  will  decide  whether 
the  Composite  Health  Care  System  increments  should  be  deployed 
prior  to  the  inclusion  of  the  archive  and  retrieve  capability. 

RECOMMENDATION  2;  The  GAO  recommended  that  the  Secretary  of 
Defense  direct  the  Assistant  Secretary  of  Defense  for  Health 
Affairs  and  the  Military  Departments  to  refrain  from  further 
deployment  of  the  system  until  completion  of  operational  testing 
and  evaluation  and  the  performance  of  a  cost/benefit  analysis 
that  justified  such  a  deployment,  (p.  12/  GAO  Draft  Report) 

POP. Response;  Concur.  The  deployment  of  the  Composite  Health 
Care  System  following  completion  of  operational  test  and 
evaluation  and  a  cost/benefit  analysis  is  consistent  with  the 
policy  of  the  Major  Automated  Information  System  Review  Council 
that  functional  capabilities  of  any  program  be  tested  and 
evaluated  adequately  prior  to  deployment  and  the  Composite  Health 
Care  System  is  no  exception.  A  full  operational  test  and 
evaluation  will  be  conducted  prior  to  deployment  of  each 
increment.  An  updated  cost/benefits  analysis  will  be  validated 
prior  to  incremental  deployment  decision  reviews  by  the  Council. 
The  deployment  decisions  will  also  consider  future  increments, 
the  capabilities  being  provided,  the  deployment  schedules,  and 
risks  associated  with  those  increments. 
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The  following  are  gao’s  comments  on  the  Department  of  Defense’s  letter 
dated  August  22, 1991. 


GAO  Comments 


1.  Defense  partially  concurs  with  our  findings  related  to  the  costs  and 
benefits  of  chcs.  We  agree  with  Defense  that  the  cost  and  benefit  anal¬ 
ysis  is  a  continuous  process.  We  further  support  the  effort  Defense  is 
making  to  validate  and  continuously  update  its  estimates.  We  have  also 
revised  the  report  to  clarify  the  reason  that  the  chcs  cost  ceiling  was 
changed. 

2.  We  are  pleased  that  Defense  believes  archiving  and  retrieving  of 
patient  records  is  essential  to  the  system’s  operation.  However,  the  bal¬ 
ance  of  the  comment  contains  both  factual  errors  and  a  mistaken 
conclusion. 

We  do  not  agree  with  Defense’s  comment  that  the  development  of  the 
archive  and  retrieve  capability  has  been  completed.  According  to  a  con¬ 
tractor  progress  report  dated  August  26, 1991,  archiving  is  still  under 
development.  In  fact,  identification  of  the  files  and  fields  to  be  archived 
is  not  expected  to  be  finalized  until  the  September  1991  reporting 
period,  and  operational  testing  of  a  fully  integrated  archiving  system  is 
not  scheduled  until  after  the  milestone  III  deployment  decision.  The  plan 
not  to  field-test  the  archiving  capability  prior  to  a  full  deployment  deci¬ 
sion  is  not  consistent  with  technology  testing  standards  or  Defense  regu¬ 
lations.  Laboratory  testing  is  not  a  substitute  for  real-life  operational 
tests.  A  maisrc  deployment  decision  on  a  $1.6  billion  patient  medical 
information  system  should  not  be  based  on  a  contractor  laboratory  test. 

The  contention  by  Defense  that  deploying  chcs  without  the  archive  and 
retrieve  function  is  low  risk  because  it  will  be  some  18-24  months  before 
this  capability  is  needed,  ignores  the  complexity  of  the  function  and  the 
essentiality  of  archiving  to  both  successful  medical  operations  and  cost 
containment.  There  is  no  guarantee  that  the  archiving  function  will  be 
operational  in  18-24  months.  Complex  software  is  often  delivered  late 
and  often  performs  badly.  We  continue  to  believe  that  because  this  capa¬ 
bility  is  so  essential  and  the  implementation  of  an  effective  solution  is  so 
complex,  it  should  be  tested  in  an  environment  that  is  operationally  real¬ 
istic  and  representative  of  the  sites,  as  outlined  in  Defense  policy.  Fol¬ 
lowing  this  path  will  reduce  risk  and  increase  the  probability  that 
Defense  will  field  a  system  that  is  operationally  effective  and  suitable 
for  the  mission  for  which  it  is  intended. 
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3.  The  completion  of  a  streamlined  method  for  physicians’  entry  of 
orders  is  critical  to  physicians’  acceptance  of  chcs  as  well  as  a  signifi¬ 
cant  purported  dollar  amount  of  benefits.  We  did  not  intend  to  make  an 
overall  comparison  in  our  report  of  the  time  it  took  for  a  physician  to 
write  an  order  with  the  overall  advantages  of  entering  that  order  in 
chcs.  Our  purpose  was  to  focus  on  the  need  to  gain  physician  acceptance 
by  improving  the  ease  with  which  physicians  can  enter  their  orders.  We 
have  revised  our  report  to  clarify  this  and  reflect  the  improvements  that 
have  been  made  in  the  entry  of  physician  orders. 

The  system,  however,  is  still  unacceptable  to  physicians.  This  is  pri¬ 
marily  because  it  is  tedious  and  cumbersome  for  physicians  to  enter  con¬ 
ditional  and  complex  orders.  The  entry  of  these  types  of  orders  can 
require  a  physician  to  enter  data  through  the  keyboard  to  bring  up  as 
many  as  10  different  screens.  Some  improvements  have  been  made  in 
the  way  physicians  enter  these  orders,  but  the  system  is  still  not  “user 
friendly"  to  physicians. 

This  lack  of  acceptance  by  physicians  was  evident  when  the  function¬ 
ality  was  deployed  to  the  test  sites  in  the  spring  of  1990.  As  of  August 
1991,  only  5  of  the  12  test  sites  had  chosen  to  activate  the  order-entry 
functionality.  Defense’s  current  schedule  does  not  yet  contain  a  date  for 
deploying  the  physician’s  one-line-order-entry. 

4.  In  our  report  we  have  emphasized  the  importance  of  one-line-order- 
entry  to  chcs  being  able  to  accomplish  its  mission,  obtain  wide-based 
physician  acceptance,  and  then  demonstrate  that  the  system  is  cost  ben¬ 
eficial.  We  continue  to  believe  that  the  risks  involved  in  deploying  chcs 
without  this  capability  need  to  be  carefully  evaluated  and  the  costs  and 
benefits  carefully  analyzed. 

The  wisdom  of  whether  to  field  chcs  without  one-line-order-entry  is  sep¬ 
arate  from  the  question  of  how  Defense  can  best  provide  for  medical 
treatment  facilities  with  “desperate  needs”  or  "aging  automated  sys¬ 
tems  that  require  immediate  replacement.”  The  Congress,  in  the  Fiscal 
Year  1991  National  Defense  Authorization  Act  established  a  process  for 
Defense  to  make  deployments  to  meet  these  needs.  This  process  requires 
Defense  to  certify  that,  among  other  things,  the  cues  software  version  to 
be  deployed  is  the  most  cost-effective  method  of  meeting  the  need,  is 
successfully  tested,  and  does  not  adversely  effect  the  contractor’s 
ability  to  complete  ongoing  ot&e.  Defense  can  make  these  deployments  if 
it  can  certify  them  in  accordance  with  congressional  direction. 
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5.  Defense  claims  that  its  early  deployments  of  chcs  did  not  violate  its 
directives  and  legislation.  We  do  not  agree.  The  appropriations  act  and 
Defense  instructions  clearly  require  the  completion  of  a  milestone 
review  and  approval  before  the  obligation  of  funds  to  acquire  a  system 
such  as  chcs. 

With  regard  to  the  appropriations  act  prohibition  on  fund  obligation  for 
major  information  systems  that  have  not  successfully  completed  over¬ 
sight  reviews  "required  by  Defense  Department  Regulations,”  Defense 
asserts  that  the  milestone  reviews  required  by  dod  Instruction  7920.2 
only  represent  the  "minimum”  set  of  review  points  for  management 
oversight.  Defense  argues  that  nothing  in  the  statutory  prohibition  on 
obligations  limits  oversight  reviews  to  milestone  reviews,  and  that  man¬ 
agement  reviews  like  the  maisrc  in-process  review  held  on  May  21, 1991, 
meet  the  statutory  language  for  fund  obligation. 

We  do  not  agree  with  Defense’s  position.  We  accept  both  the  fact  that 
milestone  reviews  are  “minimums”  and  that  management  can  involve 
itself  with  reviewing  programs  at  times  other  than  the  milestones.  Mile¬ 
stone  reviews,  however,  are  the  only  reviews  required  by  Defense. 

While  the  Instruction  authorizes  in-process  reviews,  it  does  not  require 
them  ,heir  occurrence  is  left  to  maisrc’s  discretion  and  is  a  maisrc 
requiiv.ment  only.  Moreover,  even  the  June  1988  advice  of  the  Office  of 
the  Secretary  of  Defense  specified  that  existing  legislation  requires  that 
not  only  does  a  system  have  to  successfully  complete  “an  appropriate 
management  review”  before  funds  are  obligated,  but  the  required  “mile¬ 
stone  approval”  has  to  be  obtained.  Defense’s  comments  seem  to  be 
inconsistent  with  that  advice. 

Defense  responded  that  deployments  to  date  are  consistent  with  Review 
Committee  authorizations  and  existing  legislation  because  Defense  regu¬ 
lations  specifically  provide  for  conditional  approvals.  We  do  not  agree 
with  this  position.  A  conditional  approval  does  not  necessarily  equate  to 
successfully  completing  the  statutorily  required  oversight  review 
because  the  condition  must  still  be  met  for  the  milestone  requirement  to 
be  satisfied.  Because  these  deployments  to  the  Air  Force  sites  and  addi¬ 
tional  19  military  facilities  were  made  prior  to  the  appropriate  milestone 
decision,  we  continue  to  believe  they  were  in  violation  of  Defense  direc¬ 
tives  and  legislation. 

6.  Defense  states  that  cues  was  deployed  to  Malcolm  Grow  Medical 
Center  for  certification  of  the  Patient  Appointment  and  Scheduling 
module  in  compliance  with  congressional  direction.  This  statement  is 
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inaccurate  because  deployment  to  Malcolm  Grow  was  initiated  prior  to 
the  congressional  direction  on  site  certification.  The  deployment  to  Mal¬ 
colm  Grow  was  initiated  with  a  delivery  order  dated  May  14, 1990, 
totalling  over  $70,665  to  conduct  a  site  survey .  The  requirement  to  cer¬ 
tify  deployment  of  chcs  to  additional  medical  treatment  facilities  was 
not  established  by  the  Congress  until  November  5, 1990.  At  that  time, 
Defense  had  already  issued  delivery  orders  for  over  $2.5  million  to 
install  chcs  at  the  center  and  was  preparing  to  activate  a  minimum  of 
four  chcs  modules — Pharmacy,  Patient  Appointment  and  Scheduling, 
Radiology,  and  Laboratory. 

Defense  states  that  deployment  of  a  chcs  appointment  and  scheduling 
system  was  based  on  a  sound  business  decision.  However,  chcs  program 
officials  have  not  provided  us  with  documentation  showing  the  basis  of 
this  decision  and  have  informed  us  that  they  were  not  aware  of  any 
cost-benefit  analysis  justifying  this  deployment. 

The  cost  information  presented  in  our  report  regarding  the  premature 
deployment  of  cues  beyond  established  test  sites  was  based  on  informa¬ 
tion  provided  by  Defense.  The  $5.4  million  for  deployment  to  Air  Force 
sites  is  based  on  Air  Force  documentation.  This  documentation  shows 
the  cost  of  deploying  chcs  to  58  sites  to  be  $3.8  million  in  Other  Procure¬ 
ment  funds,  the  amount  uted  in  the  Defense  comments,  and  $1.6  million 
in  Operation  &  Maintenance  funds. 

The  $7.8  million  cited  in  our  report  has  been  adjusted  to  $7.5  million  as  a 
result  of  a  reexamination  of  Defense  contract  delivery  orders.  The  infor¬ 
mation  was  obtained  from  the  chcs  Contracting  Officer  and  indicates 
that  $2.0  million  worth  of  orders  were  issued  for  conducting  site  surveys 
(including  the  site  survey  at  Malcolm  Grow  Medical  Center)  and  $5.5 
million  to  install,  operate,  and  maintain  CHCS-related  hardware  and 
software  at  facilities  that  are  not  test  sites  (the  latter  figure  includes 
$3.2  million  to  install,  operate  and  maintain  the  Patient  Appointment 
and  Scheduling  module  at  Malcolm  Grow  Medical  Center). 

7.  We  agree  with  Defense  that  the  archive  and  retrieval  capability 
should  not  be  deployed  until  all  relevant  data  are  presented  to  maisrc. 

We  continue  to  believe,  however,  that  as  stated  in  the  Agency  Comments 
section  of  our  report  and  as  detailed  in  Comment  2  of  this  appendix,  that 
this  capability  is  essential  to  the  effective  and  suitable  operation  of  any 
increment  of  chcs.  Therefore,  it  must  be  operationally  tested  at  the  test 
sites  before  the  first  milestone  III  deployment  of  chcs  is  made. 
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Marion  S.  Chastain,  Staff  Evaluator 
Anthony  Kinsey,  Staff  Evaluator 
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Medical  ADP  Systems:  Composite  Health  Care  System:  Defense  Faces  a 
Difficult  Task  (GAO/iMTEC-90-42,  Mar.  15, 1990). 

Medical  ADP  Systems:  Composite  Health  Care  System  Operational  Tests 
Extended  (gaq/imtec-89-30,  Apr.  10, 1989). 


Medical  ADP  Systems;  Analysis  of  Technical  Aspects  of  pod’s  Composite 
Health  Care  System  (gao/imtec-88-27,  Jul.  11, 1988). 

Medical  ADP  Systems:  Composite  Health  Care  System  Acquisition — 
Fair,  Reasonable,  and  Supported  (gao/imtec-88-26,  Mar.  4, 1988). 


Medical  ADP  Systems:  Composite  Health  Care  System  Operational  Test 
and  Evaluation  Costs  (gao/imtec-88-18br,  Jan.  28, 1988). 


ADP  Systems:  Concerns  About  pod’s  Composite  Health  Care  System 
Development  Contracts  (gao/imtec-87-25,  Jun.  8, 1987). 


ADP  Systems:  Concerns  About  the  Acquisition  Plan  for  pod’s  Composite 
Health  Care  System  (gao/imtec-86-12,  Mar.  31, 1986). 
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